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ABSTRACT 


This report describes the testing of four Model Output 
Statistics prediction methods on simulated data fields for 
the purpose of determining their relative skills in fore- 
casting a generic weather parameter (predictand). Of the 
four methods, three use Bayes Law of Inverse Probability to 
discriminate, while the other method uses conditional prob- 
ability. The simulated data sets, models and observers 
necessary to accomplish this goal are created according to a 
uniguely developed simulation design. The results indicate 
that there is a definite difference in the ability of one of 
the four methods, namely the méthod using conditional prob- 
ability, to forecast the weather parameter. Through the use 
of the Analysis of Variance (ANOVA) technique, this differ- 


ence is found to be significant with respect to chance. 
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I. INTRODUCTION 


The Model Output Statistics (MOS) approach to fore- 
casting consists of relating the numerical model output 
parameters, diagnostic and prognostic (predictors), to 
sensible operationally-important weather parameters (predic- 
tands), e.g. visibility, cloud amount, precipitation, for 
the purpose of enhancing the skill of forecasting these 
parameters (Glahn and Lowry, 1972). 

The first major MOS work was readied for application in 
the early 1970's by the National Weather Service (NWS), the 
weather arm of the National Oceanic and Atmospheric 
Administration (NOAA). More recently, both the U.S. Navy and 
the U.S. Air Force have been involved in the development of 
MOS schemes. NOAA'S programs are operational, both at 
civilian and military sites, with continuing development by 
the Techniques Development EAabporatony (BDL) .> = National 
Weather Service, Silver Springs, MD. 

The Navy's MOS effort began at the Naval Postgraduate 
School (NPS) in Monterey, CA in the mid/late 1970's, using 
visibility and fog over the North Pacific Ocean as the 
predictands of interest (Renard and Thompson, 1984; Koziara, 
Renard and Thompson,1983). These experiments were limited 
and the results not immediately operationally applicable. 
However, the studies formed the basis for a decision by the 
U.S. Navy to pursue the development of MOS for all oceans of 
the world, for a select number of air/ocean parameters. 
Following this decision in 1981, a series of studies were 
initiated at NPS, as a joint effort of NPS's Meteorology 
Department and the Naval Environmental Prediction Research 
Facility (NEPRF), Monterey, CA The first study (Karl,1984) 


investigated the use of three conditional probability MOS 


prediction schemes developed by Preisendorfer! (1983a, be 
using visibility at the model initialization time as the 
predictand and the output from Fleet Numerical Oceanography 
Center's (FNOC ) Navy Operational Global Atmospheric 
Prediction System (NOGAPS) model as predictors, applied to a 
limited homogeneous region of the North Atlantic Ocean. Karl 
intercompared the results of these three methods’9 and 
multiple linear regression methods with variable thresh- 
olding, as proposed by Lowe (1984). Diunizio (1984) followed 
Karl with a similar experiment, but for additional homoge- 
neous Atlantic Ocean areas, and forecast intervals to 48 
hours, and with modifications to the MOS methodologies. The 
third study (Wooster, 1984) concentrated on cloud amount and 
ceiling, using essentially the same MOS methods as his pred- 
ecessors with further variations in the multiple linear 
regression threshold model. The most recent effort (Elias, 
1985), contrasted a new model, namely the Principal 
Discriminant Method (Preisendorfer, 1984), with the earlier 
methods on their ability to predict visibility. 

This study concerns the testing of three MOS prediction 
methods exercised by the previous NPS investigations (the 
Maximum-Probability Method ale the Multiple Linear 
Regression Method, and the Principal Discriminant Method), 
plus one additional method (Discriminant Analysis Method), 
on statistically-derived simulated (ieee controlled) 
predictor/predictand data sets, with the goal of ranking the 
methodologies as to their relative skill in predictand 


Specification. 


Ipr. RLW. Preisendorfer was the Naval. Air Systems 
Command G. J. Haltiner Research Chair Professor in the 
Department of eras oe: Naval . Postgraduate School, 
Monterey, CA for 1983. Dr. Preisendorfer 1S Curznent eee 
lated with NOAA s Pacific Marine Environmental Laboratory 
(PMEL) in Seattle, WA 
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II. OBJECTIVES AND APPROACH 


There are three main objectives in this particular study 
Pomerebitine to.the U.S. Navy s MOS program. The first 
objective is to develop fields of simulated data that have 
controllable predictor/predictand parameters, such that the 
conditions of predictability of the data fields can be 
varied and the results made reproducible. The second objec- 
tive is to test four MOS prediction methods, using the simu- 
lated data fields, in order to determine their relative 
ranking in the skill of predicting the generic sensible 
weather parameter. The third objective is to determine the 
most skillful MOS method on which further testing could be 
concentrated. 

The approach taken to fulfill these objectives was to 
program the simulation procedures outlined by Preisendorfer 
(1985) so as to create the data fields. A training and three 
test sets were generated for each MOS methodology in order 
to provide ample scoring statistics upon which to analyze 
the results. For this study, two such training/testing sets 
of simulated data were generated with 1200 rows of nine 
columns, each column representing a real primary data field. 
One set was called ‘the easy data set’, while the other was 
called ‘the hard data set'. The easy data set was one that 
the MOS prediction methods could easily make a prediction 
from, due to the relatively high correlation between the 
meeatctors and the predictand. The hard data set was one 
that would have less correlation between the predictors and 
the predictand, thus making a prediction more difficult. The 
two data sets were determined in such a way as to have a 


2 


Signal-to-noise ratio~ of 4:1 for the easy data set and 1:1 


?The Signal-to-noise ratio is defined as the maximum 
ratio of the between-class distance to the within-class 


ie 


for the hard data set. Two numerical weather prediction 
models were also simulated in this study. The first is 
called the good model because it introduced less distort 
and less noise to the signal, while the bad model produced a 
greater distortion and a larger noise component. Also in 
this study, three observers were Simulated in order to look 
at the resultant variation in predictive skill of the four 
methods, as the skill of the observer varied. The three 
observers were: i) the perfect observer - one who never 
makes a wrong observation of the actual weather event; ii) 
the good observer - one who occasionally makes a wrong 
observation of the actual weather event, but not by more 
than one category; and lastly iii) the bad observer - one 
who makes a wrong observation of the actual weather event 
more often than the good observer, and sometimes by more 


than one category. 


distance of two (or more) multivariate populations given in 
the function: 


S/N = Lath) -u (2) y2) G2 (for univariate (Sibuaeionre 
where n(1) is the mean of the first category Mote tesseee 
eingy compared, 2) is the mean of the other category 
O 


and is the average variance of the two categories being’ 
compared. 


eZ 


Pie we HOPObeGlhs OF The mMOs SIMULATED DATA SETS 


The sections that follow describe the procedures used to 
create the simulated data sets, the simulated models and the 
Simulated observers for this study, as based on guidance 


provided by Preisendorfer (1985). 


A. THE MOS SIMULATION PROBLEM 

The problem of designing an MOS simulation process is 
threefold in nature. Besides designing the simulated natural 
data fields, there is. also the problem of prescribing the 
skill of the observer viewing them, and of defining how 
accurately a numerical weather prediction model would repro- 
duce them. The observer, on the one hand, would be viewing a 
parameter, such as visibility at sea, and have to estimate 
it subjectively. On the other hand, the model will be repro- 
ducing output parameters such as pressure, temperature, 
winds, etc., that could be used by any of the MOS prediction 
methods to predict the visibility, or other sensible weather 
Parameters. 

In setting up an MOS scheme, i.e., to train it to fore- 
cast the sensible weather parameter, the observations of the 
predictand must be coupled with the output parameters of the 
model. Then, when the method is to be tested, it will, 
according to its training, take a fresh model predictor set 
and produce a forecast of the predictand. Thus, in 
attempting to design a procedure uSing the simulated data 
sets, where there is built - in correlation between the real 
predictors, a simulation of the model fields must be done as 
well, including whatever biases and errors that are inherent 
to numerical weather prediction models. The ability of the 
observer must also be modeled so that it does a controllably 
imperfect job in estimating the weather parameter in ques- 


mon. This idea is illustrated in Fig. l. 
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The left-hand column represents the real atmosphere or 
ocean as given to us by nature. The real primary fields are 
those which are normally measured or are known in principle. 
Specifically, the real primary fields are those which are 
normally measured or observed and incorporated routinely 
into the Newtonian equations of motion and the Laws of 
Thermodynamics. The real secondary fields are those of fames 
casting interest in this study, since the numerical models 
do not usually forecast them directly. The right-hand column 
represents the modelled atmosphere or ocean. The model 
primary and secondary fields here are only as good as the 
model that derives them and the human that observes them, 
respectively. It is recognized that there is always some 
level of error in these fields. The MOS methods in this 
Simulation Study will take the output parameters 
(predictors) from the modelled primary field, pair them with 
the simulated observations of the secondary field, and use 
these pairings subsequently to forecast the simulated real 


secondary field parameter (predictand). 


By (tHE MOS PREDICTION CONCEPT 

Now for a more detailed look at the MOS procedure 
outlined above. A generalization of the MOS approach to 
prediction of sensible weather parameters is shown in Fig. 
l. Some estimated value of a real secondary field parameter 
(such as visibility) is recorded by an observer for some 
location at time t. This estimate is called the estaiigeem 
predictand since the secondary field is the field that is to 
be predicted uSing statistics derived from these estimates. 
Meanwhile, at the Same location and time t the numerical 
weather prediction model produces values for the model 
primary fields. The model primary fields are called the 
model predictors. 

These model predictor/estimated predictand pairings are 


taken over a large region of interest and provide what shall 
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be termed the Basic Data Set (BDS). An MOS method (X, Y, 
., Z) is then trained on a part of the BDS to forecast a 
value of the predictand field from the set of model 
predictors it is given. This method is then tested on the 
remaining part of the BDS as follows. 
1) A set of model predictors is chosen from the testing 
past of the BDS, using the predictors obtained during 
he training stage. 


2) The method then produces a forecast of the predictand 
from this set. 


3) This forecast predictand is then compared with the 
estimated predictand that was paired with the set of 
model predictor values used. 


4) Then a skill score is assigned to the forecast of this 
particular method. 


Sees rocedure is then repeated for the other MOS 
methods to be tested. 


Once all of the scores have been attained, the various 
methods are intercompared using these skill scores and some 
assessment is made as to their relative abilities to accu- 
rately predict the estimated predictand value from the given 
set of model predictors. The important assumption made here 
is that the methods will have the same relative ranking 
using the simulated data as they would have when subseq- 
uently tested with a fresh set of model predictor values 
from an actual data set of model output. This assumption 
will be well-founded provided that: 

1) The simulated data set is representative (in a statis- 
tical sense) of the real primary and secondary fields 


encountered in nature; and if 


2) The. errors of the models and observers have been 
well-simulated. 


C. SIMULATING THE REAL PRIMARY FIELDS 

The real primary fields, which may be thought of as time 
series at p fixed points in space, were generated from a set 
of pxp covariance matrices using a formulation scheme 
suggested by Preisendorfer (1985, Appendix C). The field-to- 


Prelae(7.e., cross-) correlations are represented by the 


ie 


matrix M, while the time lagged field-to-field correlations 
are represented by pxp matrix K. The scheme was developed 
with the goal of saving on computation time by defining the 
covariance matrices to be associated with stationary 
processes in space. For example, suppose m(x,x') is the 
entry of M-in row x and column x’, them the matricesuesae 
stationary, in that the covariances depend only onwieme 
differences (x-x') of the arguments x and x' in m(x,x'), so 
that it may be written as m(x-x') and k(x-x') for K. In 
this context, the eigenvalues and eigenvectors of M and K 
are expressible using simple algebraic formulas. The need 
for various matrix manipulating subroutines is’ thereby 
eliminated. 


The formula used to create the pxp Memactriceis. 
TNMs Ko) oC arene N aere |) (3. 19 


x,x' = 0, ..., p-l for 0 < » < @ , where g is the vaupapeme 
that controls the field-to-field correlation. Tite )oxouee 


Mattix- 1s formed  samilariy: 
FCG yc) he ae Ge ep ene |ae cum) (Jae) 


Sak 69 Oe ns Dl eee On es kK, *« 1, where k, is the Vanieiae 
that controls how much of a lag-induced difference there is 
between matrices M and k. (This is a simplified version of 
a more general approach in Preisendorfer (1985, Appendix 
B).) In this study, p was set to equal 9, So that theme 
series used simulated nine points in Space. 

For this study two separate data sets were formed. The 
difference between them is the amount of correlation and the 
amount of lag defined for the 9x9 matrices M and (Kye 
degree to which the set of predictors produced by the model 


were correlated and also the amount of lag-effect between 
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the pair of matrices determined whether or not the MOS 
prediction method would have an easy or a hard time fore- 
casting the sensible weather paramater (predictand). Table I 
gives an example of the covariance matrices M and K with p = 
0.15 and xc, = 0.90. These values provide a set of covariance 
matrices that are well-correlated and thus labelled the 
"easy' data set. The values for the variables of the ‘hard’ 
fea set are pg = 0.50 and ck, = 0.60. This produced a corre- 
lation that is much lower and a lag-induced effect that is 
ereater. Therefore, the forecast made from this Basic Data 
Set is more difficult. The lower half of Table I shows these 
matrices from the hard data set. 

The range of values of the x,x' pairs is especially 
tailored for the spatially stationary context, and in fact, 
arithmetic modulo p must be used on the spatial-index x-x' 
values that do not fall within the prescribed range. For 
example, if x-x' is not in the set {x: 0, ..., p-l}, x-x' 
must be reduced modulo p to map it into the finite set. The 
diagram in Fig. 2 illustrates this idea for the case p = 9. 
The range of x,x' can be visualized as being on a circle 
where the 0O,..., p-l (=8) values are plotted on it. Any 
values of x-x' outside their range are wrapped around the 
circle modulo p, so, in this way, all members of the set of 
integers (each representing the spatial location of a time 
series) can be handled. 


Ey eeOnstruction, the values of m(x,x') and k(x,x') 


depend only on x-x', and by symmetry we have m(x,x') = 
mec x) and k(x,x') = k(x',x). Hence, from now on m(x,x') = 
m(0,x-x') can be written as m(x-x'), i.e., with only one 
argument. In relating the diagram in Fig. 2 to the matrix 


Values in Table I, notice that m(8,7) = m(1), while m(7,8) =" 
m(-1) = m(8), and since, by symmetry, m(7,8) = m(8,7), it 
can be seen that m(8) = m(l). The same holds true for matrix 


K. Table II shows M and K in the simplified notation. 
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The data sets were generated using the following autore- 


gresSive equation: 


DCE. x ) = Omer y(0) . BCs) < (3 y(j) : (335) 


[b(j,t) * cos(e(j) * x) + c(i,t) S9simt Gee 


where m=4 and p=2emtl for j=0, ..., m, with teJ (the setae 
integers); MaUR 247, 7D. yGN= (GON ee K(j) = 
(Zena ip: 


The terms used in Eq.(3.3) will be elaborated on indi- 
vidually or in pairs in the following paragraphs, starting 
with the first equations derived from Eqs.(3.1) and (sm 

The A(j) term found in the expression for p(j) is the 
set of eigenvalues determined for the M matrix by the 


following formula: 


m 
NG eae OR we 05 (2 m(x) ‘cos (x(a) mee (3.4) 
x= 
with A (p-j) = At) using modular arYenmetucce and 
k(ji) = (2¢7*j)/p, for j = 1, ..., m. Through this vVatilapee 


the field-to-field matrix correlations were expressed. 


The autogressive correlations p(j) were given) byaage 


formu ra: 
m : ; 
BAG) eCON ee ee (2 k(x) « cos[k(j) * *]}7 Aes 
x= 
for j = 0, ..., m; with p(p-j) = p(j) fen 19] ieee 
again by modular arithmetic. This relation takes into 


account the lag correlation and is used to obtain theyvawa. 
ances for the random forcing of Eq. (3.3). These varniamece 


were obtained through use of the formulae: 


o°(B>i) = 2.0 © (1 - p*G)) Gre (3.6) 
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PelG ine 0 =) = p55) (1. - 6(0,4)) (3.7) 


aoreej = Off ...; m , where §(90;3) is a special case of 
hreonecker's Delta function §(i,j) Witch) i — 0), In 
general, §(i,j) = lif i=j, and =O if i + j. 

The square root of the variances obtained in Eqs. (3.6) 
and (3.7) (i.e., the standard deviations) are used in a 
random number’ generator subroutine? to get the random 
forcing terms, such that their values are normally distrib- 


uted with a zero mean, 1.e.: 
B(i,t)~N(O,07(B 55)) and y(5,t)~N(O,o7(y ,5)) (3.8) 


Observe that o*(B 0am = oy So) tone ge- Leese... m, and 
that o°(B ,0) = 4*[1-p*(0)], while o@(y ,0) = 0. 

The autoregressive correlations p(j) are used again 
along with the random forcing terms to develop the time- 


dependent coefficients b(j,t) and c(j,t) via the following 


formulae: 
met) — os) * btj,t-1) + B(j,t) | (3.9). 
meme) — o(j) © €(j,t-1) + y@,t) (3.10) 
for j = 0, ..., m.¢ 


fee sIlIMULATING THE REAL SECONDARY FIELD 
The next step in the simulation process is the modelling 
of the real secondary fields (predictands). This required 


defining a link between the real primary and secondary 


3The random number generator used is NPS's W.R. Church 
Computer Center s_ library subroutine, GGUBS, which creates 
Watbates With tniform distributions. These uniform distribu- 
tions are changed into asso an normal distributions through 
the use of the method of Box and Muller (1958). 


a . 
Nogjiee that c(0,t is uniquel zero due to the fact 
Ghat oe (y ,0) = 5 | ‘ y 


1? 


fields. The fact of the matter is that there are, in many 
cases, no readily available algorithms which can be applied 
to the real primary fields to obtain the real secondary 
fields. Indeed, this is at present a vigorously pursued set 
of problems in meteorology and oceanography. This fact is 
the basis for the need of MOS-type procedures in the first 
place! Fortunately, in this Simulation study the real forms 
of such linkages are not needed. Therefore, in order to 
provide the linkage desired, it is sufficient to invent some 
reasonable-appearing relations. Therefore, predictors could 
have been combined together in any number of different ways, 
Since the number of algebraic and analytic possibilities are 
endless; for the present study the method chosen is the 
following. There are in the set of real primary fields 
generated by Eq.(3.3), mine columns of predictor time 
series, 2Z(t,0) through Z(t,8), for t = 1, ..., 1200 sie 
1200 is the number of entries in each column. By taking one 
of these columns and relabelling it the real secondary 
field, the desired effect of having a linkage between the 
predictors and the predictands 1S assured. 

So now, by following this procedure, there is one’ 
predictand column and eight predictor columns with a corre- 
lation factor and a lag factor that can be controlled by ene 
use of the 4 and the x, terms. This method of peneraaeae 
predictors and predictand is termed the) in-Houseuiaeoe 
Method, see (Preisendorfer,1985; Appendix B). 

For this study, the predictand column chosen is the 
Z(t,0) - column because of the symmetry of the values to 
either side of it. The correlation values are highest for 
the nearest two neighbors, Z(t,l) and Z(t,8), and decrease 
in order going away from Z(t,0) in either direction. The 
correlation of the eight predictor columns to the predictand 
column is shown in Fig. 3 and Fig. 4 for the easy andwharad 


data sets, respectively. 


Zo 


Next, the predictand column is sorted by entry magni- 
tudes, with the entries of lowest magnitude being placed at 
the top of the column, while those rows with the largest 
magnitudes are placed at the bottom. At the same time, the 
other eight columns of predictors are also sorted. In this 
way the row relationship between the predictand (Z(t,0) 
Malue) and the predictors (Z(t,1)ethrough Z(t,8)) is main- 
tained. Once this is accomplished, the values in the predic- 
tand column are grouped in the following manner. 

1) For _ the upper 400 entries, those with the lowest numer- 
ical values, are tagged with the value 1 and comprise 
Se eel soececs) Ce the below tercile in 

2) For the middle 1700 entries, those between the _ two 
Me ene thereby the ‘normal’ teccile) 

3) For the _lowest 400 entries, those with the largest 
numerical values are tagged with the value 3 and 
comprise category 3 (forming the ‘above' tercile). 

The middle category, category 2, contain the most 
entries because of a desire to keep the variances for the 
three categories equal, or nearly so. This proved to be 
quite a challenge, and only after many experiments with 
various category sizes was it accomplished. Fig. 5 shows an 
example of the three categories with equally populous inter- 
vals (400 entries in each). The high narrow spike in the 
middle category indicates that the variance for that 
category was much less than either of the two categories on 
the wings, and a small variation between the training set 
and the test sets led to widely different verification 
scores. This result was unsatisfactory, and so a different 
interval size was sought that would not lead to this type of 
instability. The interval size that was settled on? was to 


have the interval size determined by the standard deviation. 


"This idea of interval s acing uSing the standard devia- 
tion as a measure was suggested by Lowe in a private discus- 
Sion, and upon testing proved to have the stability desired 
when there were at least 400 entries per predictor column. 
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The overall interval spacing for a gaussian (normal) distri- 
bution can be defined by + three standard deviations either 
Side of the mean. Hence, for this study, the middle two 
standard deviation (sigma) intervals are defined as the 
middle category, category 2, while the two outer sigma 
intervals on either side are defined as category 1 and 
CAueROTY 4 4s) Sheowne um fire eeor 

The final step for the predictand-category defining 
process is to randomly sample down from the 1/700 entries in 
the middle category to obtain the desired 1200 entry size 
for each column, which means that each category subset has 


400 values, and also nearly the same variance. 


E. SIMULATING THE MODEL PRIMARY FIELDS (MODEL PREDICTORS ) 

The counterparts to the real primary fields are the 
model primary fields, as shown in Fig. 1. The model primary 
fields are the imperfect versions of nature's real primary 
fields since they contain some distortion and noise due to 
the inability of man to model the atmosphere and ocean 
SeCeCurabte ly. 

The model imperfection is simulated in this’ study 


through the following equation: 
p-1 
X(tax) = > Stix.x' ) * Zee) 4 lee (3 sie) 
x=] 


Here the model primary field (X(t,x)) is produced first by 
having the real primary field (Z(t,x)) distorted through 
multiplication with a matrix S,(x,x ) that comsiSesaiae 
fraction on the diagonal and zero elsewhere, and second by 
having some noise added to it, element by element. The Sq 
matrices used in this simulation are shown in Table III. 
They are special cases of the more general linear transfor- 
mataons possable om the Z(t, )eeeveuce 

The value on the diagonal for the good model is 0.95 and 


for the bad model; it is 0.50. The noise Seermesm eee 


ae 


created using the same random number generator as before, 
only now different values are used for the variances to 
control the amount of spread about the Zero mean. The 
normal distributions obtained are generated with a? = Oe Ol 
for the good model, and with g% = 0.25 for the bad model. 
Thus, the good model has 95% of the original Z(t,x) value, 
plus a random perturbation from the centered normal distri- 
bution with standard deviation g = 0.1. The bad model has 
only 50% of the original Z(t,x) value, and a perturbation of 
standard deviation g = 0.5. 

F. SIMULATING THE OBSERVED REAL SECONDARY FIELDS (ESTIMATED 

PREDICTAND ) 

The final simulation requires the creation of an 
observer who will make estimates of the real secondary field 
(estimated predictand). These observations with their 
inherent errors will be combined with the model predictors 
by the MOS prediction methods and be used to forecast the 
real secondary field parameter as shown in Fig. l. 

For this study, the estimated predictand was created 
With marine atmospheric visibility in mind as the sensible 
weather parameter being forecast. So the estimates of the 
predictand are grouped into discrete categories according to 
whatever limits are desired to separate the categories. In 
this study the generic predictand (considered as marine 
atmospheric visibility) was grouped into three categories - 
[meee , 32, representing good, marginal and bad visibility, 
respectively. In making an estimation of the predictand, the 
observer may correctly choose the category of the actual 
event, or the observer may miss and choose one of the other 
two categories. For example, if there is only marginal visi- 
bility occurring at some place, for some time t, then the 
observer can choose category 2, and be correct, or category 
Ll or 3, and be wrong. A good observer would have the 


ability (or skill) to make the correct estimate most of the 


Zs 


time, but a bad observer wouldn't. Table IV Shows “aie 
table which summarizes the relative frequency e(i,j) of the 
observer estimating category i when, in fact, category j 
occurs in nature. The e(i,j) values are normalized so that 
each of the columns sum to 1. Here a perfect observer would 
have e(i,j) = 0 when i; j, and e(i,j) = 1 when 2 = J, (faa 
= 1,2,3. A bad observer would have greater off-diagonal 
values indicating an inability to observe cauregriye 

The observer's eStimate is modeled in the following 
manner. The real secondary field (predictands), described 
earlier in Section D, are arranged in ascending order 
according to their numerical values, and relabelled 1, 2, 3, 
such that the upper 400 values equal 1, the middle 400 
values equal 2, and the last 400 values equal 3. To simulate 
the observer's role a uniformly distributed number us 
chosen on the interval I = [0,1] which has been partitioned 


accordine te the observer s skit), 4 .eu 


Ale is ets. 20) on Cy = ei) (3. E2e 
A(25;j) = { u: e(1,j) < u-s e(155) 5c @eom 
A(333) =-€ ur weGl ) 46 (24a) = eee 


So. Ghat vl A(1,j) + A(2,3) + A(3,3), i = LS 2 
example, if the mumber u, chosen randomly, falls into 
A(2,j), then the category assigned will be category 2 for 
the estimated predictand when in fact category i,j = 1,2,3 
GecuEs . 

The three observers simulated in this study have skills® 
decreasing from 100% for the perfect observer, to 8/4 for 
the good observer, and still lower to 69% for the bad 


observer. Their respective skills are shown in Table V. 


oBy skills it is meant the abiltty (tomecce we Cone ae 
the actual weather event which is occurring at the time o 

observation. This ability can be defined by the calculation 
of the average of the main diagonal elements of the veri- 
fying relative frequency (i.e., contingency) table. 
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IV. THE MOS PREDICTION METHODS AND THEIR RESULTS 


A. METHOD DESCRIPTIONS 
1. Principal Discriminant Method 

The Principal Discriminant Method (PDM) was proposed 
by Preisendorfer (1984). This method is basically a discrim- 
inant method in that the data are partitioned into predic- 
tand classes and forecasts are made by forming probabilities 
for the predictor as to whether it belongs to one predictand 
class or the other. This particular method of discrimination 
is distinguished by the fact that it fits a gaussian prob- 
ability density distribution (for example) to each predic- 
tand category subset, using a principal component analysis 
of the data points in the category subset. The method's 
most novel feature is that, if the categorical distributions 
are Saseta fveant ly non-gaussian, then a successive, 
controlled splitting of the category subset is performed in 
Bae local principal component coordinate frame to obtain a 
better fitting of probability density functions. With each 
test set, fresh values.of predictors are used and the asso- 
Cliated probability density values for each category are 
found. The forecast of the predictand is then made using 
Bayes Law of Inverse Probability. 

The Principal Discriminant Method also contains a 
methodology for predictor selection. Although each of the 
four methods to be tested has its own method of predictor 
selection, the PDM was chosen to select the three predictors 
to be used by all of the methods. The basic groundwork for 
the PDM and its variable-selection process was programmed by 
Elias (1985) in three separate programs (a single predictor 


Screening program, a predictor correlation program and a 
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multiple predictor program). Hale’ subsequently made addi- 
tional modifications to these programs to more closely 
follow Preisendorfer's original formulation. 

The procedure used to Select variables and implement 
the PDM is now as follows. 

1) The initial screening program is run where each 
predictor is separately ranked according to potential 
predictability. (CPP and potential total percentage 
correct (PAO) for the training set. The first prediemen 
chosen is that with the highest PP 

2) Then the correlation program is run to find the wimesgs 
candidate predictor to be added such that it is the 
least correlated to those already chosen. 

3) The PDM multiple predictor program which implements the 
methodology described above is then run _on these 
eee eeas chosen in step 2. If the PP and PAO scores 

oth increase then the candidate predictor chosen _ in 

step 2 is added. If the PP and PAO do not increase then 
step two is repeated. The process is carried out until 
three predictors are chosen. 

4) With. the three chosen predictors, the PDM multiple 

redictor program i21S rum using the” test  seece he 


P 
Bae aa compiles a contingency table and computes veri- 
VINE sbaciseues. 


2. Maximum=Probability Metnhodml. 

The Maximum-Probability Method II (MaxProb2) was 
also proposed by Preisendorfer (1983a,b) and exercised in 
all of the previous MOS studies. It differs trom “ee 
diSeriminants methods in that the forecasts are based on 
conditional probabilities for given values of the predictor. 
To accomplish this, the data are first classed into 
n-dimensional predictor space. The discrete cells into which 
the points are placed are of a size determined by dividing 
each predictor into equally populous intervals. Then within 
each cell the number of points belonging to each predictand 
class is tallied and the conditional probabilities for each 
predictand class are formed. Thus, each specific cell has 


its own conditional probabilities and the  predictand 


’Robert A. Hale, NPS, joined the NPS/NEPRE MOS@proejeen 
team in February 1985. He has principal responsibility for 
the management of the MOS-archived data fields and Fortran 
programs. 
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category which has the maximum conditional probability is 
forecast for that cell. With each test set, fresh values of 
predictors are used to identify the cell location in the 
predictor space and the forecasted predictand category is 
returned. A contingency table of forecast versus observed 
values is then formed from which the verifying statistics 
can be calculated. 

The procedure used in this study employed _ two 
Fortran programs written by Karl (1984). 

1) First, a program is run which determines the number of 
equally Bore ous intervals into which each predictor 
should be divided. 

2) A program which implements the Maximum-Probability 
methodology for a multiple number of predictors is run 
second. In this study, the three predictors were chosen 
by the Principal Discriminant Method. 


Jen DESecri mination by DimenSion Reduction using 
Regression _ 


The DiScrimination by Dimension Reduction using 
Regression (DDRR) method was proposed by Lowe.® This method, 
in combination with various thresholding techniques, has 
been used in all previous NPS MOS studies since Karl (1984), 
Where it was called the Multiple Linear Regression method. 
The method uses the BMDP Statistical Software [University of 
California, 1983] programs - PIR, P5D and P4F. The PIR 
program carries out a multiple linear regression on the 
input predictor distributions; The P5D program displays 
various statistics (mean and variance of the classification 
functions) and histograms for the dimensionally-reduced 
estimated variable, Y, produced by the P1R program; whereas, 
the P4F program produces a multiway frequency display which 
uses the P5D output statistics with the predictor values of 
the test sets to form a contingency table from which the 


verification scores and other statistics of interest can be 


8the method was described in a private conversation and 
developed for this study based on a set of example programs 
provided by Lowe. 
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obtained. The procedure followed for this method is outlined 


below. 


1) The PIR program performs a dimension reduction sijeame 


2) 


3) 


proposed by Lowe 


ne 


linear regression. The regression equation whieneme 
derived from the predictor values of the training set 
by the PIR program is: 


= G(0)+C(1)*X(1)+C(2 eX (2) CG eam Coa) 


where C(0) is the intercept and ee C(2) and C(3 ame 
the regression coefficients. This Iinear least squares 
fit produces a new variate, Y, which contains all the 
information of the three predictors. For each test set, 
the _ respective values of 1), X(2) and X(3) preédicitem. 
will yield a fresh value o ; 


The. dimensionally-reduced Y variate iS not a2 "pages 
ability, but is used as an index or proxy in_ the 
discriminant procedures to be implemented next. The Y 
variate is grouped by the predictand categories and the 
P5P program is used to obtain the mieanm and variance 
O “Stace ruSsties sO fae ene classi fication functions 
auSsian probability density functions are fit to these 
newly formed groups via the following equation: 
L(m,n) = exp{-0.5 © [(¥ - p(m))/a2(m)]2 - (4.2) 
(= aay ica ici 
where m,n = 1, 2, 3 forming six discriminant funebioames 
Then Bayes Law of Inverse Probability is used as a 
transform statement in the P4F program, using these six 
discriminant functions to discriminate the category om 
the predictand by choosin the one with the maximum 
probability (value ieee Se through. 4.0m These 
probabilities are given by: 
P(tycs- 1 sls E21 = Gres (4.3) 
POZ) sl / (ls E).2) sees (4.4) 
P33.) = lef (Cl +. LU 3) + eee (4.5) 
4. Discriminant Analysis Method 


The Discriminant Analysis Method (DISC) was sean 
9 


and uses Fisher s classical diserimiunaue 


7 The DISC method was obtained in the same fashion as tne 
DDRR method. 
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analysis design. The procedure consists of using the BMDP 
Statistical  Sottware [University of California, 1983] 
programs - P/M and P4F. The P/M program performs a stepwise 
discriminant analysis of the input predictor distributions, 
while the P4F program is used, as with the DDRR method, to 
produce the multiway frequency table information. The proce- 


dure employed is. 


1) Run the P/7M program on the training set for the three 
redictors chosen by the PDM and obtain the classifica- 
Zane Lunettons (1, ji )aaswhere = is the index of the 

predictand category and j is the index of the predictor 
variable and the intercept. 


2) Calculate from the set of classification functions the 
set of six discriminant functions needed to perform the 
discrimination by taking the differences between the 
elements to form the coefficients. as seen in Eq.(4.6). 
Enter these values in the transform statement of the 
P4F program. 


Pie e mp pom, ) Ctr, iy pex(lj+(C(m,2)-C(ms2)] (4.6) 
°X(2)+[C(m,3)-C(n,3)]°X(3)+[C(m,4)-C(n,4)]}, 


where X(1) 2) and X(3) are the variables to be 
as by the three predictors of the test sets and m,n 


3) Then these discriminant functions are used.in Bayes Law 
Inverse Probability to calculate the probabilities 
1), P(2) and P(3) of the predictor set belonging to 
ategories 1, 2 and 3 eee oe Eqs.(4.3) through 
ee are used here also to find the probabilities. 
h 
h 


+) 


il 
O 
IE 
C 
( 
E P4F program using the discriminant functions and 
ie 

C 


test set predictors compiles a Soo ee ney table of 
the 


e 
e 
he forecast versus observed categories o redic- 
and from which the verifying scores are calculated. 


Eee RESULTS 
The successful simulation of the data, models’ and 


observers for this study is apparent from the verification 
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calculated. The perfect observer achieved the best 
scores for all the data sets (highest values for the AO and 
TSl scores, lowest values for the Al score), while the bad 


observer had the worst scores; the good model did better 


10se6 Appendix A for verification score definitions and 
comments. 
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than the bad model in all scores calculated; and the easy 
data outscored the hard data by a wide margin. These results 
are tabulated in Tables VI - VIII. 

The PDM, DDRR and DISC methods appear to have remarkably 
Similar values for the AO, Al and TSl verification scores. 
However, the MaxProb2 method has markedly different values. 
On the one hand, it scored better than the other three 
methods--higher values for AO and TS1l, lower values for 
Al--in the training set. This is probably due to the 
discrete way MaxProb2 pairs the predictand categories to the 
predictor intervals. On the other hand, it scored worse in 
the test sets. This gap between the training set scores and 
the test set scores occurs only with the MaxProb2 method. It 
appears evident that the MaxProb2 method will provide an 
extremely good fit to the training set data, if given enough 
intervals. However, if the least-squares-fit of the data of 
the testing set does not match that of the training set, 
then the verification scores show minimal skill. The other 
three methods have lesser differences between the training 


set and their test sets. 
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No. ANALYSIS OF VARIANCE (ANOVA) INTERCOMPARISONS OF THE MOS 


A. INTRODUCTION TO ANOVA 

The variations of one or more factors (prediction 
method, data set, model, observer) involved in this MOS 
study can be analyzed effectively by the technique of anal- 
ysis of variance (ANOVA). This technique allows the vari- 
ance of the measured variable (in this case, skill score) to 
be broken down into the portions caused by several factors, 
and interactions of those factors, whether varied singly or 
in combination, and a portion attributed to experimental 
error. ANOVA consists of: 

1) A. partitioning of the total sum of the ‘squares of devi- 
ations of the skill score from the mean into two or 
more component sums of squares, each of which is asso- 
ciated with a particular factor or with experimental 
error, and 


2) a_parallel partitioning of the total number of degrees 
of freedom. 


When certain variations of a factor are singled out for 
Study because they are considered to be of more importance 
or interest, then ANOVA can be used as a comparison of the 
mean effects of those certain variations. Statistical tests 
(F tests) are made to determine whether the observed differ- 
ences are probably real. If the differences are judged real, 
the main effects and interactions of the population may be 
estimated quite easily. 

The ANOVA table utilized is described below.?+ 

Mm ethe first column lists the sources of variation and 
indicates which of the sources are being varied. For 


example, source ABC has three sources of variation with 
source D being held fixed. 


Ilyor further explanation of the ANOVA printouts, see 
Box, Hunter and Hunter (1978). 
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2) The second column lists_the sums of the Squares from 
all sources listed in column comes 


3) The third column lists the number _of degrees of freedom 
associated with the sources in column one and is calcu- 
lated by subtracting one from the number of items being 
varied. 

4) The fourth, column lists the mean of the square devia- 
tions and 1s calculated by dividing the sums of squwamee 
by the degrees of freedom. 


5) The fifth column _lists the F test value observedmeaam 
the sources in column one. 


6) The sixth column lists the F test critical value which 
MS Pees es Co tbe ee ee in the fifth columns 
7) The seventh column lists the Pvalue which is the acme! 
probability that the variation observed 1s Sade uae 
chance. Thus, a very low value would indicate that the 
observed variation 1s not due to chance. 

B. THE INTERCOMPARTSONS OF ALL FOUR METHODS USING ie: 
Tables IX - XI contain the intercomparisons of the four 
MOS prediction methods using ANOVA, for the AQ, Al and TS1 
Scores, reSpectively. In these tables it can be seen that 
the variability source A (predictxvon method) 1s Signifticame 
as compared to chance since the Fobs value is greater than 
the Ferit value. Thus, there is a real difference between 
the methods that is more than just random. The large values 
evident for the Fobs terms for variance sources B, © amauee 
(data, model and observer, respectively) are due to the 
differences simulated in their generation. Of interest 1s 
the fact that the variation due to the differences in the 
data sets is nearly twice that of the variations due Eo@Eme 
differences in the model versions or the observer types. The 
negative values on the tables are due to the computer not 


properly handling very small) varies 


C. THE INTERCOMPARISONS OF THREE METHODS, MAXPROB2 REMOVED 
The large Fobs term in the preceding tables for source A 
was due to differences between the methods. As seen in 


Chapter IV, Section B, the only method that appears to have 
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markedly different values is the MaxProb2 method. The 
results of an ANOVA comparison, when MaxProb2 is removed, 
can be seen in Tables XII - XIV, for the AO, Al and TSl 
scores. It is to be noted that the Fobs terms for source A 
in each is now less than the Fcrit terms, indicating that 
there is no significant difference between the three 
discriminant methods. The data variation is still nearly 


twice as large as the other two simulated variations. 


free bos ENTERCOMPARISONS OF THREE METHODS, 2X2 

Finally, a third set of ANOVA tables, this time a 2x2 
comparison of the three remaining MOS prediction methods for 
each of the verification scores AQ, Al and TS1, can be seen 
in Tables XV - XXIII. This last comparation shows that there 
is no Significant difference between either the PDM, DDRR or 
DISC methods, for any of the verification scores, regardless 


of how the methods are grouped. 


So 


VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

First, the DDRR, DISC and PDM all have similar ski 
all variations of model, observer and data when tested on 
the simulated data. 

Second, the MaxProb2 method is not on the same perform- 
ance level as the three other methods tested, and should be 
modified in future test studies with the recommendation by 
Preisendorfer that follows (see the final recommendation 
below). 

Third, further co-evaluation of the three remaining MOS 
prediction methods should be continued with a goal of iden- 
tifying statistically significant MOS predictive schemes for 
specific forecast problems. 

Finally, the results from the Anova intercomparisons 
indicate that the variation due to the difference in the two 
data sets is the most important factor to be considered when 
seeking skillful MOS prediction schemes. Therefore, “daga 
collected must be carefully analyzed by objective methods 
(so as to identify those predictors whose values possess a 
greater separability when grouped according to predictand 
categories) in order for the resulting MOS forecast to be of 


potentially high operational worth. 


Baw RECOMMENDATIONS 

First, the predictive skill of the MOS methods, when 
predictand categories have unequal frequencies, needs to be 
addressed. For example, the rare event (one which occurs 
very infrequently, i.e., less than 104 of the time) Sis 


PareLeulac interese - 
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Second, the predictive skill of the MOS methods, when 
the predictor's class conditional populations (as determined 
by the predictand categories) have significantly unequal 
variances, also needs to be addressed. 

Third, different types of parametric probability distri- 
butions, other than the normal distribution used to generate 
the data fields, need to be developed so as to ascertain the 
relative skill of the MOS predictive methods when confronted 
with significantly non-normal data. It may be important in 
the future to relax the spatially stationary feature in the 
Simulated data equations in order to achieve this recommen- 
dation. 

Fourth, predictand formulation methods other than the 
predominantly linear In-House Field method used in this 
study need to be investigated. The scientific literature 
contains many specific algorithms, which are non-linear, 
that connect real primary and real secondary fields (e.g., 
vapor pressure, humidity, wind, etc.), these can be profit- 
' ably used in future simulation studies. 

Fifth, provide a stronger foundation for future MOS 
studies by developing new techniques for the screening of 
predictors and predictor selection. | 

ieee ene Stochasttce Skill of the MOS prediction 
methods needs to be examined. In this study, a categorical 
forecasting procedure was used where the skill statistics 
are computed from a contingency table. However, an alter- 
nate forecasting procedure would use the actual predicted 
probabilities of belonging to a given predictand category. 
With such a probabilistic method, the stochastic skill is 
defined by the sharpness of the forecasted probabilities. 
For example, consider a two-category problem where one 
method predicts a 90% probability of belonging to category 
one and 104 to category two. Then this method is said to be 


sharper and of higher predictive skill than a method which 
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forecasts corresponding probabilities of 554 and 45, for tae 
Same problem. | . 

Finally, the Maximum-Probability Method II strategy of 
choosing the category with maximum probability may not 
always be optimal. An alternate scheme would be to chooSe a 
category randomly, uSing its computed probability as a guide 
for the choice by a random number generator. Such a tactic 
would likely produce higher skill scores, overall, i.e., on 
average, than the MaxProb2 strategy. The basis for this can 
be demonstrated mathematically, using the Brier (1950) skill 
score, when the predictions are in terms of the probability 
of a category. However, it may also be described intui- 
tively. Suppose, e.g., that the three probabilities of the 
predictand are 0.2, 0.5 and 0.3 for bad, marginal and goed 
visibility for a certain realized predictor. The MaxProb2 
Strategy always directs the selection of the marginal 
category. If this strategy is followed many times, thenigane 
marginal category will be picked 100% of the time, and the 
low or high categories will not ever be picked. But the 
latter two occur 50% of the time, collectively. On the other 
hand, randomly choosing categories will allow the low and 
high categories, collectively, to be chosen 50% of the time. 
Clearly, by including the low and high categories in this 
way, a higher skill score would result. This new strategyl4 
Should be implemented henceforth in all further studies of 
the modified MaxProb2 method. 


l2this new Strategy and its justification were _ proposed 
by Preisendorfer in a private conversation. Its implications 
have not yet been verified through experimentation. 
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APPENDIX A 
STATISTICAL DEFINITIONS 


Ll. VERIFICATION SCORES 


See the table below for the following verification 


megre definitions. Total=kR + S + T+ U+V+tWt xX +Yete *Z 


A) AO - ‘A naught' score - describes the probability of 


making a correct forecast given the total sample of 
observed events (also known as_ the _ Total ercent 
Correct score). 


AO = (R + V+ 2Z) / Total 


B) Al - 'A one' score -. describes the probability of a one 


C) 


category error which is made when a forecast is one 
category away from what was actually observed, i.e., 
eee eon 2 forecast and either category 1, or veri- 
Led: 


Al = (S + U+W+ Y) / Total 


TSl - ‘Threat score’ - describes the reduction of 
threat of being surprised by a category 1 event. In 
terms of set theory, it is the intersection of category 


forecast value divided by the union of the observed 
and forecast category 1 values. 


TSlL=R/(R+S+T+U+X) 


Sample Contingency Table 
OmE so be Rov & D 
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2. PREDICTAND CATEGORIES 

The equal variance definition of the prediGctags 
categories implies that the single-trial probability of 
success of a random forecaster (the stochaster ) who is 
forecasting these categories, is not the same as that for 
the equally populous definition of the predictand catego- 
ries. The expected value of AO by the stochaster for the 
presently constructed equal-variance categories 1S Ooms 
and for the 54 upper critical value it is 0.5956 0500 
expected value of Al for the equal-variance is 0.4352, and 
the 54 lower critical value is 0.3532. 

These numbers are essential for an understanding of 
the verification scores in Tables VI (for AO) and Vi tee@eos 
Al). In particular, they tell us whether or not the 
observer, model and data sets have been simulated in a 
reasonable manner. For example, one would expect that the 
perfect observer, working with a good model and easy data, 
will obtain significantly high AO scores and significanta 
low Al scores for just about any reasonably competitive 
prediction scheme. This is borne out on perusal of Tables VI 
and VII. For instance, the perfect obServer, uSing a good 
model and easy data yields, for the PDM method in test set 
1, an AO score of 0.801, far above the 54 upper criemea 
value of 0.5956; moreover, the Al score, in this case, is 
0.199, far below the 54 lower critical value of 0.735325 
another instance for the PDM method in test set 1, a bad 
observer working with a bad model and hard data generates an 
AO score of 0.478, which is below the 54 upper critical 
value of 0.5956, and indeed less than the average (expected) 
value of 0.5136. The Al score, in this same case, is 0.389, 
which exceeds the 54 lower critical value of 0.3532. 

Table VIII for TSl cannot be as readily interpreted 
as the Tables for AO and Al. This is because the average 


value of TSl and its upper 54 critical value forsee 
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Stochaster are not readily derived for the equal-variance 
categories. They may, e.g., be worked out by Monte Carlo 
means for moderate sample sizes. For large sample sizes, 
asymptotic analytic estimates are possible, however, these 


will not be made here. 


So 
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TABLE IX 
ANOVA TABLE FOR AO - 4 METHODS COMPARED 


SOURCE SUMSQ DF 
A 0.1451 3 
B 0.4838 1 
c 0.2145 1 
D 0.4955 2 
AB 0.0014 3 
AC 0.0027 3 
AD 0.0003 6 
BC 0.0020 1 
BD 0.0306 2 
CD 0.0344 2 
ABC 0.0037 3 
ABD 0.0001 6 
ACD 0.0003 6 
BCD 0.0003 2 
ABCD 0.0016 6 
ERROR 0.0201 96 
TOTAL 1.4364 143 
CO 48.6228 MSE 0.0002 
pOvA aan Yan * Factors: 
B = 2 DATA SETS 
B = 3 OBSERVER TYPES 
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TABLE 


ANOVA TABLE FOR TS1 


SOURCE SUMSQ DF 
A 0.1663 3 
B 0.5971 l 
C 0.3342 l 
D 0.7522 2 
AB 0.0024 3 
AC 0.0048 3 
AD 0.0016 6 
BC 0.0009 l 
BD 0.0583 2 
CD 0.0627 2 
ABC 0.0056 3 
ABD -0.0001 6 
ACD 0.0004 6 
BCD 0.0005 2 
ABCD 0.0023 6 
ERROR 0.0385 96 
TOTAL DOL 143 
CO 37.9683 MSE 0.0004 
ANOVA TABLE WITH 4 FACTORS: 
= 4 MOS METHODS 
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TABLE XII 
ANOVA TABLE FOR AO - 


3 METHODS COMPARED 






SOURCE SUMSQ DF MS FOBS FCRIT ‘PVALUE 
A 0.0007 2 0.0003 1.50 3.078 0.244744 
B 0.3424 1 0.3424 1560.90 3.890 0.000000 
C 0.1423 1 0.1408 Gaqmee 3.890 0.000000 
D 0.3817 2 0.1908 870.02 3.078 0.000000 
AB 0.0002 2 0.0001 0.45 3.078 0.651507 : 
AC 0.0003 2 0.0002 0.70 3.078 0.527535 I 
AD -0.0001 4, 0.0000 -0.10 258 0.96583 
BC 0.0002 1 0.0002 0.77 3.890 0.408447 
BD 0.0245 2 0.0122 55.79 3.078 0.000000 
CD 0.0276 2 0.0138 62.85 3.078 0.000000 
ABC -0.0004 2 -0.0002 -0.90 3.078 0.434343. 
ABD -0.0002 4 -0.0001 =O 2.503 0.879029 
ACD 0.0003 4, 0.0001 Oey 2.503 0.824636. 
BCD 0.0005 2 0.0002 1.08 3.078 0.369748 
ABCD 0.0008 4 0.0002 0.96 2.508 0.456994 
ERROR 0.0158 7 
TOTAL 0.9364 107 | 
CO 38.7965 MSE 0.0002 
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ANOVA TABLE FOR AO - PDM VS DISC 


TABLE XV 


SOURCE SUMSQ DF 
A 0.0002 1 
B 0.2333 1 
€ 0.0901 i 
D 0.2572 2 
AB 0.0000 l 
AC 0.0000 l 
AD = Omooul 2 - 
BC 0.0000 l 
BD 0.0162 2 
CD 0.0216 2 
AEG) —0),0002 1 - 
ABD 0.0001 2 
ACD 0.0002 2 
BCD 0.0005 2 
ABCD 0.0002 2 
ERROR .0.0108 48 
TOTAL 0.6301 Hl 
CO 25.7722 MSE 0.0002 
ANOVA TABLE WITH 4 FACTORS: 
A= 3 METHODS, (PDM on Da SG) 
B = 3 OBSERVER TYPES 
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TABLE XVI 


ANOVA TABLE FOR Al - PDM VS DISC 


SOURCE  SUMSQ DF 
A 0.0008 1 
B 0.0825 l 
C 0.0345 1 
D 0.0349 2 
AB -0.0001 1 : 
AC 207000 1 : 
AD 0.0014 2 
BC 0.0046 i 
BD 0.0154 2 
CD 0.0158 2 
ABC 0.0002 1 
ABD 0.0002 2 
ACD 0.0003 2 
BCD 0.0035 2 
ABCD 0.0003 2 
ERROR 0.0105 48 
TOTAL 0.2047 7 
CO 8.2640 MSE 0.0002 
ANOVA TABLE WITH 4 FACTORS: 
A = 2 METHODS (PDM VS DISC) 
B= 2 DATACSEE 
B = 3 OBSERVER TYPES 
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ANOVA TABLE BOR] DSL = PDM VS DISC 


TABLE XVII 
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SOURCE SUMSQ DF MS FOBS . FCRIT 
A 0.0005 1 0.0005 Wao 3.958 
B 0.2840 il 0.2840 628.63 3.958 
c 0.1414 i 0.1414 312.96 3.958 
D 0.3942 2 0.1971 436.34 3.149 
AB 0.0000 1 0.0000 =O 3.958 
AC =ONOo Oo) l -0.0001 = Oyen 3.958 
AD -0.0001 2 Oooo One 3.149 
BC 0.0000 i 0.0000 0.03 3.958 
BD 0.0290 2 0.0145 32 ll 3.149 
CD 0.0358 2 0.0179 39.67 3.149 
ABC 0.0000 1 0.0000 0.03 3.958 
ABD 0.0001 2 0.0001 0.15 3.149 
ACD 0.0003 2 0.0001 0.29 3.149 
BCD 0.0003 2 0.0001 0.29 3.149 
ABCD 0.0002 2 0.0001 0.19 3.149 
ERROR 0.0217 48 
TOTAL 0.9072 zal 
CO 20.4221 MSE 0.0005 
ANOVA TABLE WITH 4 FACTORS: 
A= : METHODS (PDM VS DISC) 
C = 2 MODEL VERSIONS 
D = 3 OBSERVER TYPES 
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ANOVA TABLE FOR AO - PDM VS DDRR 


TABLE XVIII 


SOURCE SUMSQ DF 
A 0.0004 L 
B Oe2 237 iL 
C O70 750 1 
D Oe pak 2 
AB 0.0001 L 
AC 0.0095 1 
AD =6 0002 Ze > 
BC 0.0000 i 
BD O77 0069 2 
CD O2O1s2 2 
ABC =O 0002 U 7 
ABD -Q0.0001 Z 
ACD 0.0006 Z 
BCD 0.0008 Z 
ABCD 0.0003 2 
ERROR 0.0104 48 
TOTAL O50185 ca 
CO 25.8884 Moe a0. 0002 
ANOVA TABLE WITH 4 FACTORS: 
A = 2 METHODS (PDM VS DDR» 
B= ..2 DARA SEL 
D = 3 OBSERVER TYPES 
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ANOVA TABLE FOR Al - PDM VS DDRR 


TABLE XIX 


SOURCE SUMSQ DF 
A 0.0006 l 
B 0.0814 i 
C 0.0424 l 
D 0.0404 2 
AB -0.0001 i - 
AC 0.0006 1 
AD 0.0002 2 
BC 0.0030 i 
BD 0.0146 2 
CD 0.0113 2 
ABC 0.0001 1 
ABD 0.0003 2 
ACD 0.0014 2 
BCD 0.0049 2 
ABCD -0.0001 2 
ERROR 0.0099 48 
TOTAL 0.2110 7a 
CO 8.2898 MSE 0.0002 
ANOVA TABLE WITH 4 FACTORS: 
A= 2 METHODS (PDM VS DDRR) 
5 = 3 OBSERVER TYPES 
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ANOVA TABLE FOR TS1 - PDM VS DDRR 


TABLE XX 


SOURCE  SUMSQ DF MS FOBS FCRIT PVALUE 
A 0.0004 i 0.0004 0.90 3.958 0.370369 
B 0.2717 I 0.2717 615.67 3.958 0.000000 
E 0.1454 1 0.1454 329.46 3.958 0.000000 
D 0.4034 2 0.2017 - ~45i7mee 3.149 0.000000 
AB 0.0000 l 0.0000 =On 0s 3.958 0.769925 
AC 0.0000 l 0.0000 0.03 3.958 0.769763 
AD Oe 0002 2 -0.0001 ~Oe24 3.149 0.775460 
BC 0.0002 ie 0.0002 0.45 3.958 0.522512" 
BD 0.0313 2 0.0157 35.48 3.149 0.000000 
CD 0.0307 2 0.0154 34.81 3.149 0.000000 
ABG 7 SOL 0000 1 -0.0001 sree: 3.958 0.683467 
ABD 0.0001 2 0.0000 0.10 3.149 a 
ACD 0.0009 2 0.0004 1.00 3.149 0.399339 
BCD 0.0005 2 0.0003 0.57 3.149 0.589589 
ABCD 0.0001 Z 0.0001 0.14 3.149 0.838830 
ERROR Broo 48 
TOTAL 0.9055 7a 
CO 20.4018 MSE 0.0004 | 
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SOURCE SUMSQ DF MS FOBS FCRIT PVALUE 
A 0.0001 1 0.0001 0.55 3.958 0.483529 
B 0.0848 1 0.0848 390.08 3.958 0.000000 
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TODA C220 ZL | 
CO 8.1365 MSE 0.0002 
ANOVA TABLE WITH 4 FACTORS: 
A = 2 METHODS (DISC VS DDRR) 
D = 3 OBSERVER TYPES 
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